Assessing an Assessment Tool of Higher Education: Progress Report on Generic Skills (PROG) in Japan

Received Dec 21, 2013 Revised Jan 20, 2014 Accepted Feb 26, 2014 Assessment of learning outcomes of college students has been at issue around the world in recent years. Given that grade point average (GPA) may not be a reliable indicator for learning, some organizations have developed new types of assessment tools that measure generic skills, such as the Assessment of Higher Education Learning Outcome (AHELO) and the Collegiate Learning Assessment (CLA). In Japan, an assessment tool called the Progress Report on Generic Skills (PROG) has been introduced to measure college students’ generic skills. However, the validity of this test has hardly ever been studied. This paper thus provides an opportunity to analyze the validity of PROG through statistical analyses of the correlation between PROG scores and GPA as well as interviews with students and instructors. The findings show a gap between PROG scores and GPA. Three possible reasons are put forth as explanations for such a gap: 1) PROG does not test generic skills properly; 2) the university does not nurture and/or assess students’ generic skills that PROG measures; and 3) students make different levels of efforts for PROG and college course work. The findings of this research suggest using the results of PROG only as a reference tool that needs to be triangulated with other assessment tools. Keyword: